Sustaining Evidence-Based Practices –
What We Are Learning about the Michigan Experience
Statewide Survey: The statewide survey on organizational factors related to sustainability of the EBPs was developed from the key informant and focus group information. The statewide survey asked IPLT participants and key informants to rank items that emerged from the IPLT focus groups and key informant interviews related to organizational factors for sustainability and also included survey items that addressed additional areas critical to the sustainability of EBPs. The ranking items involved the following areas: choosing the EBPs; potential statewide solutions to challenges in sustaining EBPs; fidelity; training; staffing; and gathering and measuring outcomes. Additional questions that required a 7 point Likert response (strongly agree to strongly disagree) were also asked. The additional questions included more information on fidelity factors, training factors, gathering and measuring outcomes, geographic location, technology and awareness of EBPs.
A total of 190 individuals received this statewide survey and a total of 139 individuals completed the survey. Sixteen of Michigan’s 18 PIHPs were represented, with response rates ranging from 50 percent to 88.8 percent for each PIHP.
The key findings from this statewide survey as it relates to factors related to sustaining EBPs include:
- The top critical factors for PIHPs choosing and sustaining EBPs included consumer demand (53.2%), availability of ongoing funding from the state (15.1%), and ability to maintain fidelity to the model (8.6%).
- When ranking items related to statewide solutions to the challenges in sustaining EBPs, the respondents ranked highest the need to fund regions to select EBPs based on local needs (46.4%). This was followed by increasing consumer representation in the selection of EBPs (13%). The third solution that was supported by the respondents focused on the certification process with respondents indicating that it is important to certify the CMHPS/Region not just individual staff for EBPs (12.3%).
- Promoting adaptations of the EBPs to accommodate regional differences was rated most important to respondents in terms of maintaining fidelity to the EBPs (46.3%). Respondents also rated establishment of more flexible certification standards as the second most important solution to helping regions sustain the EBPs (35.1%).
- When considering staffing needs for sustaining EBPs, the respondents rated three items as the most important: ensuring that EBPs do not add to staff workload (26.5%), limiting the number of EBPs a CMHSP/Region is implementing at any given time (24.3%), and developing mechanisms to certify supervisors in multiple EBPs (22.1%).
- Establishing common outcome measures for EBPs was identified by respondents as the top-ranked statewide solution concerning gathering data and measuring outcomes (48.9%). The second most important solution for this area was to use cost-benefit analysis to determine the effectiveness of multiple EBPs (32.6%).
- The respondents noted the following challenges in sustaining EBPs:
- developing an effective outreach policy at the PIHP level
- finding consumers who meet criteria to participate in the EBPs
- securing clinician/staff certification in order to bill for the EBPs
- keeping up with the changes in billing procedures related to EBPs
- not being able to adapt the EBPs based on regional needs
- difficulty in monitoring external provider contract agencies for EBP fidelity
- holding trainings outside the region is a significant barrier
- need for more frequent, ongoing EBP trainings
- addressing cultural and diversity factors in the EBP trainings
- difficulty in documenting how EBPs benefit consumers
- need to reduce duplication of reporting EBP fidelity (maybe incorporate into audits and QI process)
- lack of standardized outcome measures for EBPs
- transportation issues for consumers and staff limit ability to deliver EBPs
- recruiting staff with the required State certification for specific EBPs
- funding for networking and communication technology
- need to increase onsite training opportunities (e.g. DVDs) and conferencing technologies
- When looking at differences in responses using the t-test for difference between predominately rural regions to predominately urban regions, the results suggest that for most items there are not significant differences. The following areas did show significant differences based on the rural/urban categories:
- Predominately rural regions would put greater importance than predominately urban regions on choosing EBPs based on the easiness of the certification process.
- Predominately rural regions would put greater weight than predominately urban regions on selected EBPs that do not increase staff workload.
- Predominately rural regions placed greater importance on establishing common outcome measures for EBPs when compared to the predominately urban regions.
- One-way ANOVA analyses were used to test for differences in the responses based on number of counties served by a region (single county; multi-county (2-6); multi-county (7-15). There were significant differences found across the three groups in only two areas: staff workload issues and certification of supervisors in multiple EBPs. The larger regions with more counties indicated a greater need to ensure that staff workload was not increased due to the EBP implementation. The single county region indicated a greater need to certify supervisors in multiple EBPs.
For further information about the evaluation study, please contact:
Mary C. Ruffolo, Ph.D., LMSW
University of Michigan School of Social Work
1080 South University
Ann Arbor, MI 48109
(734)936-4799
mruffolo@umich.edu
The study was funded though a Medicaid Match grant mechanism and was awarded to the University of Michigan Department of Psychiatry and the School of Social Work. Dave Neal, LMSW and Karen Milner, M.D. are the principal investigators of the overall grant mechanism. Mary C. Ruffolo, Ph.D. and Brian Perron, Ph.D. were the primary investigators of this evaluation study on sustainability of evidence-based practices. Stephanie Benson, MSW was the project coordinator. The University of Michigan School of Social Work Curtis Center also funded portions of this study.
Alyson Rush from the Department of Community Mental Health served as the state liaison to this study.
